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FROM THE DESK OF M. KAREGA RAUSCH, 
NACSA VP OF RESEARCH EVALUATION 


| had the privilege of authorizing charter schools in Indiana for over a decade. Our still young 
profession—only 25 years old—has the humbling and exciting responsibility of determining 
the quality of charter schools in every city and state. 


How do we deliver on that responsibility? That is the single most important question 
authorizers spend time on. | know we did. 


Our field has been well-served by the foundational practices established through the wise 
counsel and guidance of some of the nation’s first authorizers and trailblazers. Yet, the 
quality of charter school portfolios still varies tremendously from authorizer to authorizer. It’s 
clear the profession needs to know more, dig deeper, and perhaps most importantly, connect 
practices to results for students and communities. 


Fortunately, we are now at a point where there are authorizers and communities with very 
high-performing charter schools. What are the nation’s best authorizers doing differently to 
achieve great outcomes? 


This question was at the heart of NACSA’s three-year research project to identify authorizer 
practices associated with high-quality school portfolios. We launched the Quality Practice 
Project (QPP) by examining the practices of authorizers with the strongest sets of charter 
schools in the country, as measured by many student and community outcomes. We then 
looked at the practices of authorizers with average portfolios. When we compared the two, 
we were able to see clear distinctions. 


Here, we release the first results of this groundbreaking research. We learned a lot. Among 
the most important is that while successful authorizers are grounded with smart systems 
and tools, they are empowered to make the best decisions for children through great 
leadership, institutional commitment, and strong professional judgment. 


| urge you to spend some time with our key findings, the practices that separate authorizers 
with exemplary from average portfolios, and the case studies of five great authorizers. 


Of course, this work is never done. NACSA will work with authorizers and the field to 
incorporate these findings into their day-to-day work. In addition, since the needs of 
students, families, and communities continue to evolve, so will what we need to know about 
authorizing practices that meet those needs. 


We are excited to share the results of this first-of-its-kind research project on charter school 
authorizing. We are confident that this work will, over time, lead to stronger results for 
students and communities across the country. 


Sincerely, 


M. Karega Rausch, Ph.D. 
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ABOUT THE PROJECT 


The Quality Practice Project (QPP) is NACSA’s multi- 
year research initiative to identify what high-performing 
authorizers do to achieve stellar student and public 
interest outcomes. Participants and researchers 
collaboratively investigated the perspectives and practices 
of high-performing authorizers compared with a sample of 
authorizers achieving moderate, but not weak outcomes. 
This first-of-its kind research and NACSA’s report based 
on its findings are an important step toward developing a 
more evidence-informed connection between practices 
and outcomes. 


Selecting Authorizers Based 
on Outcomes 


NACSA enlisted the talents of a diverse and well- 
recognized group of authorizers and accountability 
experts to develop a set of outcome criteria used to judge 
authorizers. This Advisory Panel agreed on a set of 11 
indicators separated into two broad categories: 


Portfolio Outcomes (e.g., student performance, 
protecting student and public interest, transparency) and 
Performance Outcomes (e.g., closing failing schools, 
allowing quality schools to expand). 


Studying Similarities and Differences 


Intensive case studies were conducted with authorizers 
with strong and moderate outcomes. This involved 
document reviews, artifact reviews, multi-day on-site 
visits, interviews with authorizing institution members, and 
interviews with other key stakeholders in the authorizer’s 
context (e.g. charter associations, school leaders). Areas 
of commonality and difference were examined and 

tested, resulting in a list of practices and contexts that 
appear similar and different across these two groups of 
authorizers. 


Validation and Engagement 


Initial findings were presented to the Advisory Panel 

and participating authorizers, who provided feedback. 
Additional non-participating authorizers (through NACSA’s 
Leaders Program) and internal staff were given the 
opportunity to provide feedback on findings. Those 
engagement and feedback opportunities resulted in a 
strengthened and validated set of findings. 


Using Findings 


NACSA and authorizers with strong portfolios will work 
with the field to promote adoption of QPP practices and 
contexts for successful authorizing. In addition, NACSA 
will work with authorizers to more deeply understand 
which practices and approaches seem to make the 
strongest contributions to improved outcomes. 


The method and process used by the QPP has been used 
successfully in other educational and business research 

to identify correlates of success. It does not, however, 
yield causal relationships. Thus, findings from the QPP are 
best positioned as strong correlates of outcomes worthy 

of consideration, adoption, and further testing. More 
research is needed to yield causal relationships between 
authorizer practices and outcomes. 


A list of the QPP Advisory Panel members, QPP Sample 
& Methods, descriptions of Portfolio and Performance 
Outcomes, and the Interview & Document Review 
Protocol are all available in the appendices. 


Nacsa 
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KEY FINDINGS 


Great authorizers—those with strong school portfolios and performance outcomes—implement 
foundational best practices that NACSA has promoted for years. But to achieve outstanding 
outcomes, more is needed. When compared to others nationally, great authorizers also share 
certain additional unmistakable characteristics: 


1 LEADERSHIP “We authorize strong schools that not only 
Le Great authorizers are dedicated to a mission of create a love of learning, but actually ensure 
sess? | giving more children access to better schools SH gecic lesan. 2? A pers recente meee 


through the proactive creation and replication 
of high-quality charter schools and the closure 
of academically low-performing ones. 


that program, we replicate it. When schools 


fail to live up to their mission, they close.” 


—Susie Miller Carello, Executive Director, SUNY Charter Schools Institute 


¢ The State University of New York’s 
Charter School Institute (SUNY) sent 
a clear message early in its existence with several high-profile closures of failing 
schools, which signaled it would put the interests of students above all else and 
that trustees and staff were serious about upholding standards. These early moves 
helped improve the quality of their portfolio in another way: stronger charter school 
operators who value tough but supportive oversight have flocked to them, while 
those desiring to fly under the radar or not interested in strong accountability have 
tended not to apply to SUNY. 


¢ Metropolitan Nashville Public Schools created a successful, well-vetted 
application process that is the key to the quality of its portfolio. With smart 
application and opening tools in place, the authorizer was able to successfully 
recruit both homegrown operators that grew into networks and national charter 
management organizations to the district. It was also able to remain focused on 
quality during a period of increased application activity that followed sweeping 
education reforms in Tennessee in 2009-11. In addition, their Office of Charter 
Schools advocates for charter schools within the district, helping them navigate 
intra-district issues and ensuring that schools receive the resources and support to 
which they are entitled. 


3 JUDGMENT 
Great authorizers make decisions based on what will drive student outcomes, not based 
dudsment | On checking boxes or personal beliefs. 


e Leaders and staff at the Massachusetts Department of Elementary and 
Secondary Education and its Office of Charter Schools and School Redesign 
have created a strong culture of professional judgment. Despite having created 
many of the processes and tools now regarded as best practices in charter 
authorizing, key application and intervention decisions—to a remarkable extent— 
are grounded in the professional judgement of staff. The highest value is often the 
collective wisdom of an experienced and highly-skilled team, whose understanding 
of quality is well aligned and routinely fine tuned. As a senior staff member put it, 
“Authorizing isn’t paint by numbers.” 


e When collecting key accountability data, Washington, D.C.’s Public Charter 
School Board allows schools to correct erroneous data, even if the deadline 
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has passed. This ensures that high- 

stakes accountability is based on accurate 
information. They engage in a holistic 
approach to reviewing new and expansion 
charter applications, using a balanced 
assessment of strengths and weaknesses of 
leadership, academic program, finance, and 
equity that a scoring rubric would not. Board 
and staff join together for a final evaluation 
called “defense day,” where they debate what 
the decision should be. They have built a 
strong procedural foundation while building 
flexibility and discretion into decision making. 


“Outcomes in authorizing matter: you have 


to know whether, and to what extent, you’re 


impacting student outcomes and changing 

lives. Specifically, are the resources—time, 
money, people, professional development— 
substantially changing the education 


landscape for the better?” 


—Kathryn Mullen Upton, Vice President of Sponsorship & Dayton 
Initiatives, Thomas B. Fordham Foundation 


COMMITMENT 

Cm Great authorizers reflect their institution’s commitment to quality authorizing. Authorizing is 
visible, championed, and adequately resourced, rather than buried in a bureaucracy. The people 
responsible for day-to-day authorizing functions have influence over decision making. 


Commitment 


Sample Organizational Charts: 


Authorizer with 
High-Performing Portfolio 


Authorizer with 
Average Portfolio 


BOARD BOARD 
Uotes on charter decisions Votes on charter decisions 
AAA 
HEAD OF 
AUTHORIZING Makes recommendations to Board 
FARIA cccommencitions 
to Board 
2 Authorizing is visible, championed, and 
adequately resourced, rather than buried HEAD OF 

Mm in a bureaucracy. The people responsible AUTHORIZING 


for day-to-day authorizing functions 
have influence over decision making. 


Commitment 
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The State University of New York (SUNY) Charter School Institute’s commitment 
to expanding quality options has been fostered by its structure of political 
accountability. The Institute is not part of the State Education Department, home of 
New York’s other statewide authorizer. The Institute’s leadership reports directly to 
SUNY’s Trustees (the final decision makers on high-stakes authorizing functions and 
decisions), providing important, direct influence on these decisions. The governor 
appoints the Trustees to seven-year terms, which provides the Institute with a 

layer of electoral accountability, as well as some insulation from influences and 
agendas unrelated to the Institute’s mission. This political independence gives the 
organization a kind of nimbleness. 


Authorizing is one of three ways the Thomas B. Fordham Foundation works 

to increase school quality. The Foundation has maintained this organizational 
commitment to quality authorizing since 2005 by supporting—financially and in terms 
of human capital—a largely autonomous, highly visible authorizing operation. While 
day-to-day decisions are left to the authorizing staff, Fordham has an experienced 
and engaged leadership team (which includes head authorizing staff) and board of 
trustees. Their input helps to guide staff work and is critical to high-stakes decision 
making. Fordham’s commitment goes beyond quality authorizing of great schools: it 
seeks to be an exemplary authorizer for others to emulate. 


Nacsa 
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PRACTICES THAT MATTER 


NACSA examined the contexts and practices of authorizers with some of the strongest 
portfolios of charter schools in the country and those of authorizers with average portfolio 
performance. When we compared the two groups of authorizers, many distinctions 
emerged. 


The following contexts and practices apply to authorizers with strong charter school 
portfolios. Although there are clear distinctions that set authorizers with strong portfolios 
apart from those with average portfolios, findings associated with both cannot be 
dismissed as unimportant. Our research did not examine authorizers with low-performing 
schools; while not tested, some findings may be necessary for at least an average school 
portfolio, or foundational for a strong portfolio of schools. 


For ease of discovery, we’ve organized our findings into four categories: 

1. Authorizer culture and characteristics 

2. Application and school opening 

3. Monitoring and intervention 

4. Renewal, expansion, and closure 
NACSA encourages authorizers to review these findings against their own work and ask 
questions about how they could improve. In some settings, authorizers may encounter 
barriers preventing the replication of some of these practices and contexts. In these 


instances, authorizing institutions, authorizing staff, local advocates, and policymakers 
should work together to remove such barriers. 
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AUTHORIZER CULTURE AND CHARACTERISTICS 


Decision Making and Culture 


GOAL SETTING 


Strong Portfolios Only Both Strong and Average Portfolios 


e Authorizer goal setting and planning tends to be e N/A 
focused on annual core authorizing activities and 
are not part of a long-term (multi-year) traditional 
strategic plan. 


e Authorizers tend to have goals and activities 
specific to cyclical authorizing functions (e.g., 
application season, renewal season, site visit 
season). 


e Authorizing goals and key activities are updated at 
least annually and are specific to local context. 


e Authorizers have an intentional goal alignment 
process, in order to avoid conflicting goals and 
activities and/or to ensure they have adequate 
capacity to execute multiple goals. 


DATA “OBSESSED” CULTURE 


Strong Portfolios Only Both Strong and Average Portfolios 
e Actively and intentionally acquire school e Use accumulation of evidence to answer key 
data, including collecting data that may be questions. Authorizer never relies on one piece of 
3 disconfirming to perceptions. Data is actively data/evidence. 

J explored, ana incomersied Bue decision * Authorizing staff provides comprehensive, data- 

— making (aligned with key decisions that need based rationales for unambiguous high-stakes 
to be made, used to make all high-stakes recommendations to board. This is designed to 
decisions). 


provide the board with clear direction, information, 
e Only ask schools for information they are and justification to make high-stakes decisions. 
unable to reliably and accurately get from 
other sources. 
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DECISION MAKING 


Strong Portfolios Only 


e Professional staff is not bound by protocols, 
templates, or other authorizing tools that 


; limit their decision making. Staff has a clear 
J belief and orientation that such tools assist, 
ee not dictate, decisions (a high degree of 

professional judgment is used in decision 
making). 


¢ Staff do not use professional judgement 
in decision making based on opinions 
or beliefs, or independent of tools and 
protocols. Rather, they make decisions 
grounded in facts, data, and expertise from a 
robust body of evidence. 


e¢ Create and use protocols and processes that 
allow for nuanced discussions, and collect 
numerous qualitative and quantitative data to 
inform and justify decisions with evidence. 


e¢ Do not require or prefer an “if-then” type of 
intervention system post school opening 
(i.e., a system that lays out an authorizer’s 


5 response/requirements to specific offenses/ 
J issues that come up a priori). Rather, 
Judgment preferred intervention system includes 


professional judgment, deliberative 
conversations, and utilizing past responses 
that were effective in remedying situations. 


Both Strong and Average Portfolios 
e Have well-developed protocols and tools that are 
used in decision-making. 


e There are no “point totals” that solely dictate 
decisions. 


RELATIONSHIP TO SCHOOLS 


Strong Portfolios Only 


e Authorizers view role as supporting school 
success, not as a “compliance cop.” Yet they also 
draw a very clear line between providing “support” 
and “direction,” the latter of which is strongly 
avoided. 


e — Intentionally develop relationships with school 
staff and leadership, typically through visits to 
the school and phone calls, outside of formal 
accountability processes. 


Both Strong and Average Portfolios 


e Authorizers have a sense of humility about their 
work in relationship to that of people in schools. 
This sense of humility results in an orientation 
that authorizers not only shouldn’t, but can’t, give 
schools direction on how to improve. 
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CONTINUOUS CREATION, IMPROVEMENT, AND DISSEMINATION 


Strong Portfolios Only Both Strong and Average Portfolios 
e  Avidly seek out new/best practices from other e Entrepreneurial in creating new authorizing 
authorizers (and at times other sectors), and modify practices or new ways of executing existing 
to fit their context. authorizing practices. 
e Structured, cyclical opportunities for staff reflection e “Open-source” culture; share practices with other 
and self-critique on practices and systems. authorizers. 


¢ Continually (typically annually) review policies and 
procedures, and roll back unneeded paperwork or 
compliance burdens on schools. 


Mission and Environment 


MISSION STATEMENT AND PURPOSE 


Strong Portfolios Only Both Strong and Average Portfolios 


e N/A e Authorizers have a mission statement 
specific to authorizing that speaks to 
the organization’s unique strengths or 
circumstances. Authorizers that exist within 
a larger “parent” organization also have an 
authorizing-unique mission statement. 


e Mission and vision statements are varied, 
likely connected to local context. 


e Authorizers are principally focused on 
authorizing strong schools, but also 
: explicitly see their role as larger than that 
Le function (e.g., filling community gaps/ 
needs, catalyzing systemic change in public 
education, revolutionizing authorizing). 
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ORGANIZATIONAL VALUES 


Strong Portfolios Only Both Strong and Average Portfolios 


¢ Organizational values are explicit and e N/A 
reinforced through a range of activities, 
i such as being posted in visible locations 
throughout the office or staff readings and 
— discussions about original and current 
authorizing purposes. 


e Authorizers actively share and inculcate staff 
in their “creation story” in order to influence 
and shape current culture (i.e., how they 
became an authorizer and important early 
actions that continue to reinforce their values 
and principles, such as early closure of low- 
performing schools). 


EXTERNAL ENVIRONMENT 


Strong Portfolios Only Both Strong and Average Portfolios 


e N/A Authorizing often exists in a supportive charter school 
ecosystem with: 


e Astate association (or equivalent) focused on 
state level advocacy and that is often aligned with 
authorizer needs around quality (although some 
healthy tension was noted in many places). 


¢ Support organizations that help with application 
and new school development, often explicitly 
aligned with the authorizer’s written expectations 
(application areas and pre-opening spaces). 


e¢ Support organizations that help with existing school 
improvement. 
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Leadership and Staff Development 


AUTHORIZING WITHIN LARGER PARENT INSTITUTIONS 


Strong Portfolios Only 


2 Authorizing is a highly important and 
Cm visible function within the larger “parent” 
Eermtinert organization. 


e Role of authorizing is explicitly mentioned 
in the larger parent organization’s strategic 
plan. 


Both Strong and Average Portfolios 


e Non-authorizing staff within larger organization is 
encouraged to participate and add value to key 
authorizing activities (e.g., applications); however, 
decisions on which non-authorizing staff to include 
reside within authorizing senior leadership. 


SENIOR AUTHORIZING LEADERSHIP 


Strong Portfolios Only 


Highly empowered in decision making. 
Examples include: 


Authorizing head (e.g., Executive Director) 
Cm reports to Board and/or authorizing staff 
Pecan makes high-stakes recommendations directly 
to the Board. 


e History of long-tenured senior leadership, 
including multiple long-tenured executive 
directors (typically 5+ years). 


e Senior authorizing leadership has needed 
exemptions from “parent” organization 
policies, if any, in staff hiring, development, 
and termination. 


e Authorizing office is not subject to 
institutional forced placement (formally or 
informally pressured). 


Both Strong and Average Portfolios 


e Seta tone of urgency, provide a buffer from political 
and other distractions, and build bridges to external 
sources of information and support. 


¢ Clear conceptual agreement between senior 
authorizing staff and board on purposes of 
authorizing and decision-making alignment. 


BOARD RELATIONS 


Strong Portfolios Only 


Both Strong and Average Portfolios 


e Board subcommittees help staff with high-stakes e N/A 
decisions including serving as a thought partner 
and facilitating larger board decision making. 
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STAFF DEVELOPMENT 


Strong Portfolios Only 


3 Staff develop explicit strategies to ensure 
J a shared understanding of and expertise in 
dvasment_| quality authorizing. Examples include: 


New hires are trained in more than just 
their role (i.e., cross-training on other 
authorizing functions). 


Cross-team work (“all hands on deck” 
work) is explicit and required, and leads 
to “bench strength.” 


Authorizing staff development is a critical 
organizational commitment and tends 

to be developed around the needs of 
individual staff members aligned with 
organizational goals. 


Staff attend and present at major 
conferences. 


Specific and concrete steps are taken 
to ensure alignment among staff (e.g., 
retreats, re-norming exercises), and 
avoid decisions by anecdote or opinion. 


Both Strong and Average Portfolios 


Authorizing staff, not larger parent organization, 
drives professional development. 


Parent organization provides relevant professional 
development opportunities typically outside of 
authorizing, but are still relevant to staff (e.g., 
trainings on new state laws/policies, human 
resources). 


Authorizers have multiple examples and an 
organizational culture of “promotion from within.” 


The number of staff dedicated to authorizing 
generally and to specific authorizing functions 
varied across authorizers, but each ensured 
enough staff to manage core authorizing functions. 


Nacsa 
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Application and School Opening 


APPLICATION DEVELOPMENT 


Strong Portfolios Only Both Strong and Average Portfolios 


e Authorizer provides denied applicants 
detailed feedback to (a) provide a public 
record of why an applicant was denied and 

‘ (b) assist the applicant in reapplying in a 
later cycle. Denying an applicant (with clear 
=— feedback on the reasons for that denial) 


is not seen as a negative outcome for the 
authorizer. 


e When conditional approval is granted, its 
purpose is to specify technical changes to 
the proposal that need to be made, not as 
a method to allow the applicant to further 


View application process as an “iterative” process. 
It is not uncommon for a denied applicant to 
reapply in a future application round. 


Applicants are encouraged to contact the 
authorizer for informal conversations regarding the 
application process. Authorizers may also provide 
formal applicant training specific to the steps in 
their application process and common errors prior 
applicants have made. They do not, however, 
provide evaluative feedback on any individual 
application prior to submission. 


Authorizer has a multi-stage process in which 


develop and improve their proposal. applicants are provided feedback at each stage 


and are permitted to respond to feedback during 
the process. 


e Authorizer has an application amendment 
process and/or awards conditional approval to 
strong applicants, allowing some minor additional 
development prior to opening. 


e Authorizer gives the applicant the opportunity to 
withdraw the application prior to a formal notice of 


denial. 
TRANSPARENCY 
Strong Portfolios Only Both Strong and Average Portfolios 
e Authorizer provides applicants and the public e N/A 
detailed information about the application process 
including timelines, evaluation criteria, previously 
submitted and reviewed applications, feedback 
and correspondence with prior applicants, and 
recordings of board meetings and application 
hearings. 
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APPLICATION STAFFING 


Strong Portfolios Only 


Specific authorizing staff are assigned to oversee 
the application evaluation process. 


Authorizing and/or other parent organization 
staff members responsible for access and equity 


questions/issues are involved in application review. 


Reviewers receive detailed training on the 
application criteria and “normed” regarding what 
constitutes a successful application. 


Both Strong and Average Portfolios 


Staff from across the larger parent organization 
participates in application evaluation process (i.e., 
not just the staff assigned to “authorizing”), at the 
direction of senior authorizing staff. 


Multiple reviewers evaluate each application. 
Reviewers are trained on the application criteria. 


Application reviewers come from different 
professional backgrounds and have diverse 
expertise, but are not necessarily “external” to the 
authorizer. 


Nacsa 
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APPLICATION CRITERIA, DUE DILIGENCE, AND SCORING 


Strong Portfolios Only 


3 


Jg 


Judgment 


3 


Jg 


Judgment 


All application requirements have associated 
evaluation criteria and are formally evaluated. 


The application only includes elements 
necessary to evaluate the quality of the 
application. 


Evaluation criteria describe both the rigorous 
standard and the specific information 
required to meet the standard. 


Authorizer focuses on the strengths and 
weaknesses of each applicant and reaches 
an evidence-based recommendation 

via discussion, debate, and professional 
judgment. 


Authorizer ensures that all parts of the 
application are internally coherent and 
reinforcing. It does not evaluate an 
application solely by its ability to meet 
standards in the discrete areas of education, 
business/finances, and organizational 
capacity. 


Authorizer does not have stated preferences 
for certain school missions or types of 
educational models. The authorizer may 
identify geographic areas or communities 

of educational need, but does not specify a 
preference for specific types of schools. 


Has different requirements depending on the 
type of application received (e.g., start-up, 
replication), in-portfolio versus out-of-portfolio 
replications, type of school proposed (é.g., 
virtual, Alternative Education Campus [AEC]) 
and who is involved in the application (e.g., 
charter management organization [CMO], 
education management organization [EMO], 
independent). 


Both Strong and Average Portfolios 


Authorizer has systems to conduct due diligence on 
the performance of existing operators. 


Reviewers do not simply “score” the application, 
but identify strengths and weaknesses in each 
application. 


Require applicants to demonstrate community 
outreach and demand for the school. 
Demonstrating outreach and demand can be 
done through a variety of mediums including 
community hearings, surveys, and other sources 
of evidence of demand presented within the 
written application. Pointing to less than adequate 
academic performance among similar schools in 
an area is not enough to demonstrate demand and 
insufficient in demonstrating community outreach. 
All authorizers saw value in outreach and demand, 
although some noted it’s also in statute and not 
discretionary. 
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APPLICANT INTERVIEW 


Strong Portfolios Only 


Authorizers do not interview all applicants, but do 
interview all “qualified” applicants based on pre- 
existing standards established by the authorizer. 


Authorizers are systematic and formal about 
developing interview questions. Questions are 
developed ahead of the interview, are based 

on a thorough review of the written application, 
are coordinated across interviewers to eliminate 
redundancy, and are often scripted. 


Interview team looks for both the content of the 
answers and also who answers the question. They 
look for and evaluate the degree to which there 

is broad understanding of the application, issues 
raised, and if appropriate people are answering 
key questions (e.g., if the CMO/EMO is answering 
questions that board members should answer). 


APPLICATION DECISION MAKING 


Strong Portfolios Only 


Final submitted (and approved) application is a 
detailed blueprint for school opening and operation. 
Very little is left for later development. 


Both Strong and Average Portfolios 


The applicant interview is an essential component 
of the application evaluation process. 


Authorizer has specific “red flags” (that vary across 
authorizers) that indicate that an applicant group 
lacks the capacity to operate a school. 


Interview questions are prepared ahead of time, are 
tailored to the applicant, and designed to gather 
more information about application weaknesses or 
areas of the application that lack clarity. 


Interviews are in-person. Multiple members of the 
applicant group are asked to attend the interview 
and multiple authorizing staff members participate 
in the interview. 


Both Strong and Average Portfolios 


Authorizer relies on successive stages and 
multiple sources of information to reach application 
decisions, and applications can be denied at each 
stage. 


Authorizing staff submits recommendations for 
approval to the Board, but makes denial decisions 
without board input. 


Nacsa 


Leadership, Commitment, Judgment: Elements of Successful Charter School Authorizing 19 


PRE-OPENING PROCESS 


Strong Portfolios Only Both Strong and Average Portfolios 
e Unlike other areas of authorizing practice, e Authorizer has a pre-opening process to identify 
authorizers are very hands on (sometimes whether a school has demonstrated it is ready to 
quite intensively) in the pre-opening process, open. 
inelnding directing Schools -Oniateas for * — Authorizer isn’t afraid to hold schools accountable 
Le change, collaborating with school support for not successfully completing pre-opening 
Lecoerahie organizations, providing explicit informational 


process, including not letting a school open. 
and step-by-step resources for schools, 


and advocating on behalf of schools when 
necessary. 


e Authorizers use the pre-opening process 
to build relationships, set expectations, and 
provide technical assistance to schools. 


CONTINUOUS REFLECTION AND IMPROVEMENT 


Strong Portfolios Only Both Strong and Average Portfolios 


e Authorizer reviews its application process after e N/A 
each cycle to improve its efficiency and validity. 
The authorizer typically makes small technical 
updates to its application process after each cycle, 
and when reviews suggest a need for substantial 
changes, makes such changes after that cycle. 


e Authorizer seeks input from staff, reviewers, 
applicants, and the community regarding changes 
to the application process. 
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Monitoring and Intervention 


MONITORING 


Strong Portfolios Only Both Strong and Average Portfolios 


1 


Ongoing monitoring is clearly aligned with 
contract/charter expectations. Schools know 
exactly what the authorizer is monitoring 
and why. Similarly, because authorizer 
intervention is aligned with contract 
expectations, schools knew what things 
would and would not lead to authorizer 
intervention (although how an authorizer 
intervenes/responds varied; see section on 
use of professional judgment). 


Authorizers require detailed plans, of which 
every part is analyzed, for student enrollment 
processes and systems. Plans are typically 
approved annually, but the authorizer collects 
data to identify issues more frequently, 
typically monthly or quarterly. 


Authorizers have internal authorizing staff 
with an explicit focus on equity and access 
(although this was typically not the staff 
members’ only responsibility area). Specific 
to ongoing oversight/monitoring, such 

staff is responsible for school monitoring, 
performance framework data collection, and 
renewal decision-making information. 


Frequently collect data/monitor (monthly or 
quarterly) and have internal capacity to tell 
the difference between concerning financial 
performance data (e.g., data that might 
trigger a conversation with or notice to the 
school) compared to serious short- or long- 
term financial crises (e.g., data that required 
an immediate response from the school and/ 
or immediate authorizer action). 


Authorizers have conversations with schools when 
any issues are identified prior to issuing any formal 
notices (and many don’t issue formal notices 
unless circumstances are dire and/or school is 
unresponsive) 


Authorizers used a number of different methods 
to collect and hold schools accountable for 
organizational oversight. While an individual 
authorizer typically does not use all methods, a 
list of commonly used methods includes meetings 
with schools, written inquiries, stakeholder surveys 
and focus groups, attending governing board 
meetings, online monitoring systems in which 
schools uploaded data, site visits, and an annual 
report from schools that included reporting on 
organizational performance indicators. 


Alignment between staff operating the authorizer’s 
monitoring and intervention system (with a 
preference for the same person or people 
managing both). Such a system provides schools 
with a monitoring-intervention process that seemed 
to increase the chances that corrective actions 
were aligned with findings from ongoing monitoring. 


Nacsa 
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CHARTER FOCUS AND AMENDMENTS 


Strong Portfolios Only Both Strong and Average Portfolios 


e NA e¢ Contracts/charters are limited in scope and focused 
on only two things: (a) what is required by state 
law and (b) what is necessary to hold a school 
accountable as determined by the authorizer. 


e Authorizers set a high bar for charter amendments, 
reserving approval for changes only to areas 
deemed “material.” In addition, authorizers 
intentionally limit “material” areas that require 
formal authorizer approval to maximize school 
autonomy. 


SCHOOL FEEDBACK AND SITE VISITS 


Strong Portfolios Only Both Strong and Average Portfolios 


e Astrong feedback loops exist between the e Authorizer publishes, at least annually, individual 
authorizer’s monitoring system and schools, which school performance reports aligned to framework 
gives schools clarity on where they stand relative expectations on at least academics, operations, 
to authorizer expectations. and finances. 


e Authorizers provide schools with feedback 
documenting areas of strength or concern through 
a variety of mediums (site visit reports, informal 
conversations, compliance reports) immediately or 
shortly after monitoring activities occurred. 


e Use formal site visits to collect information about 
schools, and use the site visit process to facilitate 
difficult conversations with schools when needed. 
Information from site visits are used to provide a 
more robust assessment of school performance, 
and often augment and amplify other quantitative 
performance information. Data collected during site 
visits are intentionally and specifically planned, and 
site visits are not organized as a “gotcha” exercise. 
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PERFORMANCE MEASURES & EXPECTATIONS 


Strong Portfolios Only Both Strong and Average Portfolios 


e N/A e Performance measures are legally binding in some 
way, meaning they can be legally enforced via an 
accountability plan or performance framework. 
Those measures are typically included as part of 
the contract between an authorizer and school 
(or incorporated by reference) or other means by 
which the authorizer had legal standing to enforce 
performance expectations. 


¢ Schools can establish school specific goals that are 
approved by the authorizer, but authorizers varied 
in their encouragement of schools establishing their 
own indicators. 


e Academic accountability frameworks typically 
include measures of student growth, student 
proficiency, post-secondary indicators, college and 
career readiness indicators and school specific 
goals. Authorizers establish a common set of 
required academic goals (and, as noted previously, 
schools can set additional goals if they choose to). 


e Financial accountability measures include both 
near-term and sustainability measures. 


e In addition to governance, other common 
operational areas included facilities requirements, 
requirements to adhere to applicable law, 
requirements for special populations (i.e., reporting 
requirements and adherence to applicable law), 
enrollment process compliance and results, 
reporting and compliance requirements, student 
health and safety expectations, requirements for 
the school environment, teacher and staff licensing 
requirements, and financial management reporting 
requirements. 


e There is variance between authorizers on the 
degree to which financial and operational goals are 
uniformly applied or school-specific. 
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Charter Renewal, Expansion, & Closure 


RENEWAL 


Strong Portfolios Only 


e NA 


EXPANSION & REPLICATION 


Both Strong and Average Portfolios 


Academic performance expectations for renewal 
represent the bar for a quality school, not the 
minimum expectations to avoid closure. 


Ensure schools are not held accountable for 
expectations that are not present or known 
throughout the charter term (i.e., “no surprises’). 


To ensure renewal decisions are unambiguous, 
authorizers have clear alignment of renewal 
documents, renewal criteria, renewal rubrics, 
renewal application ratings, performance 
frameworks, charter contract clarity on renewals, 
and recommendations. 


Strong Portfolios Only 


3 


Jg 


Judgment 


1 


Authorizer’s criteria and standards for school 
operator past performance is exceptionally 
clear. Schools seeking to replicate or expand 
know if they should even apply or not. 


Replication application is not automatically 
approved, even for schools that meet past 
performance criteria and standards. The 
review for potential replicators is different 

but never automatic and never without a 
thorough review. Decisions for replication are 
based on a number of factors (e.g., capacity 
to replicate, potential location), but are most 
heavily weighted on past academic, financial, 
and organizational performance. 


Authorizers provide incentives for replication 
or expansion (e.g., reducing per-student 
oversight fee and expedited application 
process, charter amendment process rather 
than new or expedited application process, 
access to facilities). 


Both Strong and Average Portfolios 


In-portfolio replicators (i.e., those already in the 
authorizer’s portfolio) have an expedited application 
review process. 


Potential replicators, particularly in-portfolio 
replicators, are not required to submit any 
information the authorizer already possesses or 
can easily acquire. 


Nacsa 
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CLOSURE 


Strong Portfolios Only Both Strong and Average Portfolios 


e When a school’s performance meets e N/A 
the authorizer’s standards for closure, 
authorizing staff prefers to work with the 
school’s board to relinquish the charter 
rather than initiate a formal closure by the 
authorizer. 


e Authorizer informs the school and its board 
of underperformance years in advance of the 
end of the school’s charter term. Through 
multiple feedback loops described in other 
sections, including formal face-to-face 
meetings with the school leader and school’s 
board, the authorizer ensures the school is 
aware of performance that may lead to non- 
renewal, typically multiple years in advance 
of the school’s renewal cycle. 


e Authorizer takes an active role when a 
school is closed. This can include trying 
L to find a replacement operator and project 
te managing (either directly or through other 
organizations) the process of ensuring 
students have access to another school. 
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ELEMENTS IN ACTION: FIVE CASE STUDIES 


At the heart of QPP is the close-up study of a small group of stellar authorizers, chosen 
for the strong outcomes of their portfolios of schools. By spending significant time on 
the ground with these five authorizers—studying, observing, questioning, learning, 
cataloging—NACSA has significantly expanded knowledge about what great authorizing 
looks like. 


The process involved document reviews, artifact reviews, multi-day on-site visits, interviews 
with authorizing institution members, and interviews with other key stakeholders in the 
authorizer’s context (e.g., charter associations, school leaders). The result is tangible 
examples of the authorizer practices associated with high-quality charter school portfolios. 


What follows are snapshots of each of the five case studies, which are available in full on 
our website. Each sketch highlights examples of three essential elements in action in that 
particular authorizer’s office: leadership, commitment, and judgment. 
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DISTRICT OF COLUMBIA PUBLIC CHARTER SCHOOL BOARD 


“We have learned over many years, across many states and cities, that good authorizing 
is more likely to result in a strong charter school sector. NACSA has for years defined 

Dc the basic building blocks of good authorizing. The QPP project takes that a step further, 
looking beyond basic practices to more foundational conditions, such as the beliefs, 


pc. Bublie chatter | approaches, and organizational structures that characterize strong charter authorizers.” 


—Scott Pearson, Executive Director, District of Columbia Public Charter Schoo! Board 


Why DC PCSB? 


The District of Columbia Public Charter School Board (DC PCSB) is one of the top charter 
school authorizers in the country, based on an 11-point evaluation of school portfolio and 
authorizer performance outcomes. Some facts of note about DC PCSB: 


e DC PCSB opened its first charter school in 1998 and is the sole charter school 
authorizer in the District of Columbia. By 2017-18, it had grown to be authorizer of 
120 charter schools and 46% of public school students in the district were attending 
charter schools. 


e DC PCSB holds its charter schools accountable for their performance, closing 23 
schools over the past six years. 


¢ DC PCSB actively encourages high-performing schools to grow. Those efforts appear 
to have had their intended impact: during the last four years, the DC PCSB approved 
the expansion of 31 high-performing schools. 


Leadership, Commitment, Judgment at DC PCSB 


Leadership: DC PCSB approves schools to open in neighborhoods that desperately 

need good schools, even in the face of pressure to not open new schools. They minimize 
personal preference in models and approaches by creating policies and frameworks that 
Leadership narrowly focus on student academic achievement expectations, equity of access and 
learning, and financial stability. The authorizer has pioneered robust transparency in school 
results, including the areas of discipline and finances, despite pushback from schools about 
potential damage to their image. They have been rigorous about closing low-performing 
schools, even in the face of political pressure not to. 


Commitment: The DC Public Charter School Board has a single mission: to provide high- 
quality public charter schools to students and families. DC PCSB’s board is independent 
and committed to their job of quality authorizing, while maintaining a larger vision to serve 
Commitment as a national model for authorizing. The authorizer has exclusive control over the use of 
funds, and consistently invests in systems to improve authorizing. For example, they now 
have a data team to allow staff to better focus on school results and outcomes. 


Judgment: When collecting key accountability data, DC PCSB has always allowed schools 


3 
J to correct erroneous data even if the deadline has passed. This ensures that high-stakes 


accountability is based on accurate information—more important to them than adhering 
Judgment to data submission deadlines. They engage in a holistic approach to reviewing new and 
expansion charter applications that uses a blended and balanced assessment of strengths 
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and weaknesses of leadership, academic program, finance, and equity that a scoring rubric 
would not. Board and staff join together for a final evaluation called “defense day,” where all 
staff who have participated in the review gather to debate what the decision should be. They 
have built a strong procedural foundation while building flexibility into decisions, allowing 
leadership some discretion in decision making. 


IMPACT 


DC PCSB has closed 23 schools over the past six years and their results continue to 
significantly outpace district averages. Despite an oversight fee 1/3 that of many authorizers, 
they are amply funded. Having control over their resources allows them to be efficient and 

at the same time direct resources where needed. Their poorest neighborhoods have over 

a dozen high-quality “Tier 1” schools (a marker of quality in the DC community) and more 
students in Washington, D.C. attend these schools than ever before. DC Equity Reports are a 
national model in data transparency about school performance in multiple areas. 


They are not afraid to take risks in approving great, yet unproven, ideas, and indeed have 
approved some excellent schools that might have failed a strict rubric. One example is 
Monument Academy, which serves foster students and is becoming a national model in this 
area. 


The full case study, A Look at District of Columbia Public Charter School Board: Case 
Study Analysis for the Quality Practice Project, is available on our website at 
http://www. qualitycharters.org/research/quality-practice-project/. 
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THOMAS B. FORDHAM FOUNDATION 


“Outcomes in authorizing matter: you have to know whether, and to what extent, you’re 
impacting students and changing lives. Specifically, are the resources—time, money, people, 
FE professional development—substantially changing the education landscape for the better? 
NACSA’s in-depth work to identify practice-linked outcomes is important to the profession 
pone see onda because it identifies key characteristics of top authorizers. This enables others in the field 
to capitalize on this information and in turn improve the sector as a whole.” 


—Kathryn Mullen Upton, Vice President of Sponsorship & Dayton Initiatives, 
Thomas B. Fordham Foundation 


Why Fordham? 


The Thomas B. Fordham Foundation (Fordham) is one of the top charter school authorizers 
in the country, based on an 11-point evaluation of school portfolio and authorizer 
performance outcomes. Some facts of note about Fordham: 


e By 2017-18, Fordham was authorizing 13 brick and mortar charter schools located 
throughout Ohio, educating just over 5,000 students. 


e Fordham charter schools have full autonomy within the bounds of federal and state 
law. Fordham does not exert direct control or any undue influence over their schools’ 
governance, operations, or educational plans. 


e¢ The majority of its charter schools were in the “Very High” or “High” academic 
growth categories in Reading and Math at the time of NACSA’s analysis. 


Leadership, Commitment, Judgment at Fordham 


Leadership: Fordham has other functions besides authorizing; however, the authorizing 

work has never been subordinate to the other work of the organization. Fordham leadership 
and board recognize that the organization can have a positive and significant impact via the 
Leadership authorizing work, and they have supported that growth, due mainly to the replication and 
expansion of existing, high-quality charter school models. Fordham values risk-taking, a 
sense of urgency, and a commitment to excellence at all levels. It tries to be crystal clear in its 
communication with schools. 


Commitment: Fordham has made an institutional commitment to quality authorizing 

since 2005, supporting this work financially and in terms of human capital. While day-to- 

day decisions are left to the authorizing staff, Fordham has a thoughtful, experienced, and 
Commitment | @ngaged leadership team (which includes the head of Fordham’s authorizing shop) and board 
of trustees whose input helps to guide staff work and is critical to high-stakes decision making. 
Fordham’s commitment goes beyond quality authorizing of great schools: it seeks to be an 
exemplary authorizer for others to emulate. 


Judgment: Fordham believes that box-checking should never drive decisions; rather, 


3 


decisions should be made based on how students will best be served. They have created 
J solid accountability mechanisms, but decisions ultimately rely on professional judgment. 
Judgment Staff use trigger areas more than rubrics. These trigger areas are performance indicators 


showing significant change in a school, which staff then investigate. They have no formal 
process for intervention. What makes it work? The caliber of personnel, teamwork, respect, 
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and experience. Another example: Ohio’s law requires authorizers to provide technical 
assistance to their schools. Fordham takes a conservative view of what assistance is 
appropriate and doesn’t sell services to schools. They see school leaders as the experts, 
and respect school autonomy. If schools are producing the required outcomes, Fordham 
stays out of their way. 


Impact 


Fordham’s institutional commitment to quality authorizing is reflected in the success of its 
schools, seen in trustees’ decisions to approve expansions and replications of existing, 
high-performing school models. These models are then able to serve more students, which 
translates to a broader impact on successful student outcomes. One model has expanded 
from a founding class of 56 students to over 1,500 (and growing) today. This school 
perennially posts some of the highest student growth outcomes in the state. 


Leadership is dedicated to a mission of giving more kids access to a great education, 
evident in Fordham’s portfolio growth from 2,700 students in 2005, to 5,000 students in 13 
schools (and growing) in five Ohio cities today. 


The full case study, A Look at Thomas B. Fordham Foundation: Case Study Analysis for 
the Quality Practice Project, is available on our website at http://www.qualitycharters.org/ 
research/quality-practice-project/. 
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MASSACHUSETTS BOARD OF ELEMENTARY AND 
SECONDARY EDUCATION 


“The Quality Practice Project has been a great opportunity to work with others in the 
authorizing community, using an evidence-based lens to take stock of authorizing 
Mi a practices across the nation. Education leaders and stakeholders must be willing to think 
differently and creatively in order to stay relevant and effective. In Massachusetts, our 
heat Seconawy ba. authorizing practices—and judgment in how they are used—help us navigate the creative 
tension between accountability and autonomy.” 


—Cliff Chuang, Senior Associate Commissioner for Educational Options, 
Massachusetts Department of Elementary and Secondary Education 


Why BESE? 


The Massachusetts Board of Elementary and Secondary Education (BESE) is one of the top charter 
school authorizers in the country, based on an 11-point evaluation of school portfolio and authorizer 
performance outcomes. Some facts of note about BESE: 


e By 2017-18, BESE authorized 82 charter schools serving almost 45,000 students (4.5% of state 
enrollment). 


e¢ BESE provides the public accurate and transparent information about the performance of its 
schools with annual performance reports available on its website. 


e Massachusetts had more charter schools and students in “High” or “Very High” academic 
growth categories in NACSA’s analysis, compared to schools and students in “Typical” growth 
categories in more than two academic years across both English/Language Arts and Math. 


Leadership, Commitment, Judgment at BESE 


1 Leadership: BESE leaders and staff are passionate about continuous improvement, 

| knowledgeable and thoughtful about charter school development and accountability, and integral 
e to all relevant decision making. They feel responsible for maintaining the state’s strong reputation 

Leadership as the inventor of many charter oversight best practices. One example of their leadership 

in action: They recruit and approve applications from developers with the capacity to create 

high-quality schools intending to serve disadvantaged students. They do this through targeted 

recruitment of successful operators; supporting and training new developers; and ensuring 

developer capacity through a rigorous application process. 


2 Commitment: Massachusetts demonstrates an institutional commitment to quality authorizing. 
C As the state’s sole authorizer, BESE has a mission to strengthen the Commonwealth’s public 
mM education system so that every student is prepared to succeed. The Office of Charter Schools 
Commitment and School Redesign (OCSSR) reports to the Commissioner of Elementary and Secondary 
Education. During BESE meetings devoted to charter issues, staff have direct interactions with 
BESE’s appointed members, where they answer questions and debate the merits of granting, 
amending, or renewing a charter. History shows strong alignment between BESE’s decisions and 
staff recommendations. 
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J Judgment: BESE’s authorizing staff is long-tenured. They work closely together and have an 
unusually reflective practice, devoting time each year to reviewing and adjusting norms and 
sali sa judgment criteria. They also periodically test their ratings of schools against other outcomes to 
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ensure that their judgments are holding up. The longevity of staff, promotion from within, 
and direct experience with charter schools have been huge assets. This has provided 

for important leadership stability, both within the OCSSR and with charter schools in the 
Commonwealth, and has allowed managers to have a deeper knowledge of the work of 
their direct reports because they themselves have also performed these tasks. According 
to one staff, “Nobody does work alone, and nobody is doing a job that one of us has not 
done.” 


Impact 


BESE has created an environment where charter schools are thriving. The authorizer 
employs clear, transparent, rigorous, and measurable criteria. Charter applicants and 
existing schools understand the criteria and acknowledge that BESE has established 
rigorous expectations. Without this comprehensive work to authorize charter schools 
through a transparent and cohesive system, it is likely that the quality of charter 
applications and schools would decline. BESE ensures fidelity to the vision of charter 
schools as a way to offer more quality education options to Massachusetts children. 


The full case study, A Look at Massachusetts Board of Elementary and Secondary 
Education: Case Study Analysis for the Quality Practice Project, is available on our 


website at http://www. qualitycharters.org/research/quality-practice-project/. 
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METROPOLITAN NASHVILLE PUBLIC SCHOOLS 


“The profession of authorizing has developed in many different ways over the last 20 plus years. 
The QPP is incredibly important as it seeks to link practices to outcomes and identify those 


N cl practices that seem to result in high-quality educational opportunities for all students. While the 
project has not yet yielded a causal relationship between authorizer practices and outcomes, 
Metro. Nashville there are enough successful correlates to draw conclusions that there are indeed best practices 


Public Schools 


and policies that, if applied, result in excellent opportunities for all students.” 


—Carol A. Swann, Coordinator of Charter Schools, 
Metropolitan Nashville Public Schools 


Why Nashville? 


The Metropolitan Nashville Public Schools (MNPS) is one of the top charter school authorizers in the 
country, based on an 11-point evaluation of school portfolio and authorizer performance outcomes. 
Some facts of note about MNPS: 


e When the Charter Schools Office formed, MNPS was the authorizer of three existing and two 
soon-to-be-open charter schools. By 2017-18, MNPS was authorizer to 29 operating schools, 
and had closed four poor-performing schools. 


e They have strongly encouraged those schools that serve students well to replicate, and as a 
result have several CMOs with as many as five schools in their network. 


e The majority of Nashville charter schools demonstrated high academic growth across both 
Reading and Math with very few charter schools in the “Low” or “Very Low” academic growth 
categories of NACSA’s analysis. 


Leadership, Commitment, Judgment at MNPS 


Leadership: After Charter Office leaders chose to partner with NACSA on a Formative Authorizer 
Evaluation, staff participated in NACSA’s federally-funded work to study and develop exemplary 
policies and procedures relating to performance management, replication, and closure. Those 
Leadership rigorous authorizing policies and procedures shaped this strong charter sector. The Office’s 
leadership sees themselves as a catalyst for change—including within the district. Collaborative 
but impatient, the Office feels a direct responsibility toward students that is its touchstone in 
relation to both the district and the schools it oversees. 


Commitment: Since the inception of the Charter Office in 2009 to the time of our study, MNPS 
has been solely focused on providing students with more access to high-quality educational 
opportunities. At the time of our research, the office was an integral part of the district, and 
Commitment staff made recommendations directly to the board. Many of the authorizer’s policies have been 
adopted statewide, and they are proud of their strong relationship with the state department of 
education. The district’s last two strategic plans, MNPS Achieves (2009-13) and Education 2018: 
Excellence for Every Student (2013-18), have had a significant focus on transforming district 
operations and positioned charter schools as a district priority. 


3 Judgment: The Office has created a successful, well-vetted application process that is key to the 
J quality of their portfolio. With respect to oversight decisions, there are processes on the books 
for probation, but the actual decision to refer something to the Board is discretionary, relying 
ouOEMEGE heavily on the Director’s professional judgment as to whether a particular infraction merits a 
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frank conversation or raising the issue to the level of Board attention. The Office of Charter 
Schools advocates for charter schools within the district, helping them navigate intra-district 
issues and ensuring that schools receive the resources and support they are entitled to. 

In a similar situation, some authorizers might dictate prescriptive mandates infringing on 

a school’s autonomy, or not help schools navigate the complex and at times contentious 
relationships with district staff who hold unfavorable opinions of charter schools. Not so 
Nashville’s charter office, which serves as an important bridge between schools and other 
district offices. 


Impact 


Nashville sees the raising of expectations and increasing quality opportunities as 
contagious. The district is significantly increasing outreach to and inclusion of its charter 
schools, creating partnerships to benefit students district-wide. Two examples: Newcomer 
Academies, a district initiative to serve the district’s high population of students whose 

first language is not English, are housed in charter schools; and one of the authorizer’s 
charter partners opened its summer coding academy to all interested district students. The 
authorizer intends to expand these partnerships: they think a school’s governance model is 
of no consequence if that school is instrumental in facilitating high student achievement. 


Nashville believes that strong oversight, closing poorly performing schools, and encouraging 
replication of excellent ones, go together to create a robust charter sector. If any of those 
pieces are missing, the result is likely schools that do not move the needle for students 
academically, who are operationally deficient, and who are fiscally unsound. Those schools 
undermine public confidence in charter schools, and public education as a whole, not to 
mention the damage done to students. 


The full case study, A Look at Metropolitan Nashville Public Schools: Case Study Analysis 
for the Quality Practice Project, is available on our website at http://www.qualitycharters.org/ 
research/quality-practice-project/. 
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THE STATE UNIVERSITY OF NEW YORK 


“Our focus is on giving parents strong choices. We authorize strong schools that not only 
create a love of learning, but actually ensure students learn. If still more parents want that 
N program, we replicate it, because that gives parents more strong choices. When schools 
fail to live up to their mission, they close. The Institute’s mission? #moregreatseatsforkids” 


SUNY — Susie Miller Carello, Executive Director, SUNY Charter Schools Institute 


Why SUNY? 


The Charter Schools Institute at SUNY is one of the top charter school authorizers in the country, 
based on an 11-point evaluation of school portfolio 
and authorizer performance outcomes. Some facts of note about SUNY: 


e SUNY authorized its first charter school in 1999 and had 185 authorized schools by 2017-18. 


e SUNY’s portfolio of charter schools are fully accessible to all students including students of 
color, low-income students (those receiving free or reduced lunch), students with disabilities, 
and English language learners. 


e SUNY holds its charter schools accountable for their performance: 9% of SUNY-authorized 
charter schools have not been renewed by the SUNY Board of Trustees. 


Leadership, Commitment, Judgment at SUNY 


Leadership: SUNY sent a clear message early in its existence with several high-profile closures of 
failing schools, which signaled it would put the interests of students above all else and that trustees 
and staff were serious about upholding standards. These early moves helped improve the quality 
Leadership of its portfolio in another way: stronger charter school operators who value tough but supportive 
oversight have flocked to them, while those desiring to fly under the radar or not interested in strong 
accountability have tended not to apply to the Institute. SUNY also shows leadership by replicating 
high-performing schools of all types. The Institute’s portfolio contains many independent charter 
schools as well as many CMOs that started with a single school. Once authorized, SUNY schools 
receive leeway to make changes to their educational programs without significant bureaucratic hurdles 
but with the explicit understanding that they will still be held accountable for measurable student 
achievement. SUNY strives to be a nationally recognized repository and disseminator of research, 
training, and best practices for charter schools, public school choice, and authorizing. 


2 Commitment: Institutional commitment is reflected in how the Charter Schools Institute is situated 

within the SUNY system. Unlike many other higher education authorizers, the Institute is not a sub- 

Cm unit of another department within SUNY. Institute leadership reports directly to the Charter Schools 

Commitment Committee, comprised of university trustees and charged with high-stakes decision-making. From the 
Institute’s origins, the commitment to expanding quality options has been fostered by its structure of 
political accountability. The governor appoints SUNY’s Trustees to seven-year terms. The arrangement 
provides the Institute with a layer of electoral accountability, as well as some insulation from influences 
and agendas unrelated to the Institute’s mission. This political independence gives the organization an 
important nimbleness in its operations and decisions. 
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3 


Jg 


Judgment 


Impact 


Judgment: SUNY has some of the best-developed charter oversight architecture of any authorizer in 
the country, and many of its systems have been used by other authorizers. But it also has to make tough 
decisions. The Institute has built judgment capital through explicit efforts to develop the professional 
expertise of its staff, complemented by a fairly long average staff tenure at the time of the study. Part of 
the onboarding process is giving new staff an opportunity to work in all phases of authorizing. Among the 
key tenets of that system are crossfunction training. Staff strongly believe that new hires need to have 
more than just knowledge of their job and responsibilities. It is equally important for staff to understand 
how their role fits into the larger authorizing functions. Staff accomplish that goal by actually doing parts 
of the work of other teams and positions and observing key actions such as Charter Schools Committee 
meetings and closure hearings. SUNY also has a deep commitment to staff professional development 
sending staff to conferences nationally related to charter schools and other educational areas. 


More than 80% of SUNY’s charter schools provide measurably stronger educational choices than the districts in which 
they are located. SUNY proudly points to college acceptance videos, get out the vote flashmobs, modern day 
Shakespeare music videos, chess champions, culinary programs, community organizing, disaster relief—in addition to 
academic success—as some of the results SUNY-authorized charters produce for children. SUNY also offers strong 
parent interest as another proofpoint: 100,000 parents a year apply to SUNY-authorized charters. 


The full case study, A Look at State University of New York (SUNY) Charter Schools Institute: Case Study Analysis for 
the Quality Practice Project, is available on our website at 
http://www. qualitycharters.org/research/quality-practice-project/. 
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CONCLUSION 


Charter school authorizing is, ultimately, an intensely human endeavor that should be 
grounded in good laws and policies, sound principles and standards, and—day-to-day— 
smart processes, rubrics, and benchmarks. 


But by getting closer and looking more carefully at the authorizers whose portfolios of 
schools are highly successful, we can begin to understand the connection between this 
success and certain contexts and practices. These contexts and practices are shared by 
the strongest authorizers, and the rest of the field should pay close attention to them. 


Our nation’s strongest authorizers create environments where charter schools thrive. 
They help charter schools live up to their fullest potential. The best authorizers are doing 
certain things differently to achieve great results within their communities. NACSA will 
continue to share, test, and refine these findings and collaborate with authorizers around 
the country to apply them on the ground and grow the successes of our nation’s public 
charter school sector. 
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APPENDIX 


A. Advisory Panel Members 


e Alison Bagg, Massachusetts Department of Elementary and Secondary Education 
¢ Scott Bess, Purdue Polytechnic High School 

e Susie Miller Carello, SUNY Charter Schools Institute 

¢ Cliff Chuang, Massachusetts Department of Elementary and Secondary Education 
e Al Fan, Minnesota Comeback 

e Gail Greely, Formerly with Alameda County Office of Education (now retired) 


e Ethan Hemming, Formerly with the Colorado Charter School Institute (now out of 
authorizing) 


¢ Bonnie Holliday, State Charter Schools Commission of Georgia 


e¢ Mark Modrcin, Formerly with Tulsa Public Schools (now with Nevada State Public 
Charter Authority) 


¢ Tiffanie Pauline, Miami-Dade Public Schools 

e Scott Pearson, DC Public Charter School Board 

e Greg Richmond, National Association of Charter School Authorizers 
e Marcus Robinson, Memphis Education Fund 


e Margo Roen, Formerly with the Tennessee Achievement School District (now an 
independent consultant) 


¢ Elliot Smalley, South Carolina Public Charter School District 
e Carol Swann, Metropolitan Nashville Public Schools 

e Kathryn Mullen Upton, Thomas B. Fordham Foundation 

e Robbyn Wahby, Missouri Charter Public School Commission 
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B. QUALITY PRACTICE PROJECT SAMPLE & METHODS 


METHODS AND CRITERIA FOR PORTFOLIO AND PERFORMANCE OUTCOMES 


NACSA enlisted the talents of a diverse and well-recognized group of authorizers and 
accountability experts to identify and define key outcomes, as well as important areas to 
probe as a part of the case study process of the Quality Practice Project (QPP). This Advisory 
Panel agreed on a set of 11 indicators that can define quality authorizing outcomes. These 11 
outcomes are separated into two broad categories of Portfolio Outcomes and Performance 
Outcomes. 


The full list of Portfolio Outcome and Performance Outcome measures, including how each 
was assessed, is included as Appendix C at the end of this document. In general, schools in 
an authorizer’s portfolio were assessed on the following areas: 


1. Proportion of high-performing schools; 
Proportion of low-performing schools; 
Financial viability; 

Socio-demographic representation; 

Ethical resource management; 

Availability of school performance information; 
Extent of autonomy in decision making; 


Extent of strong new school openings; 


oO momNnN oa FF WO YD 


Closing schools with egregious academic, operational, financial, or unlawful practices; 
10. Closing schools with very poor academic performance; and 


11. Expansion of schools with strong academic performance. 


Due to data availability, resources, and time, some of the assessment methods used are 
indirect proxies of the outcome(s) of interest. While the measurement method of each 
individual outcome indicator is imperfect, the combination of strong performance across 
indicators provides greater confidence that selected authorizers are authorizers worth 
studying. 


AUTHORIZER INCLUSION CRITERIA 
Given the need to narrow the scope of the project to authorizers of most interest, the Advisory 


Panel provided guidance on initial selection criteria. Those criteria included authorizers that: 


e Chartered at least seven schools. This was a proxy measure for capturing 
authorizers who see chartering as part of a meaningful change strategy and nota 
“one-off” program they engaged in; 

e Were in operation for at least seven years. This criterion was established in order to 
examine the track record of high-stakes decisions and practices; and 


¢ Opened or closed at least one school over the past five years. This criterion 
ensures that potential authorizers are active in authorizing practices. 
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The Advisory Panel indicated that while smaller authorizers—especially those with only 
a few schools—are important to learn from, the types of practices they adopt are likely 
fundamentally different than authorizers with some level of scale. The Panel agreed that 
smaller authorizers should be studied separately and not part of the initial QPP. 


Application of the three inclusion criteria yielded 103 potential authorizers for inclusion (using 
data from 2015). 


The Advisory Panel strongly encouraged NACSA to focus most heavily on student growth 
outcomes relative to student proficiency. There were eight states that did not have a growth 
model or accessible growth information despite multiple data acquisition attempts in 2015. 
Those eight states are included in Figure 1. For all authorizers in states without growth 
information, proficiency rates were examined using the same targets as authorizers with 
growth information. 


FIGURE 1: STATES EXAMINED VIA PROFICIENCY RATES ONLY 


Proficiency Only 
N Arkansas 


hh , California 


< Connecticut 
nd Louisiana 


a Maryland 
| New Hampshire 
) & New York* 


Oklahoma 


*Note: We were able to obtain growth information for the State University of New York (SUNY) 
but unable to obtain those data for other authorizers in the state of New York. 


AUTHORIZER SELECTION METHOD: STRONG PERFORMANCE GROUP 


The Advisory Panel indicated that not having an academically strong portfolio of schools was 
a “deal breaker’—meaning strong academic performance was a requirement for inclusion 

in the group of authorizers with strong outcomes. Given the primacy placed on student 
academic outcomes, and student growth in particular, outcomes on Indicators 1 and 2 were 
assessed first. Twenty-two (22) authorizers met the targets for both Indicator 1 and Indicator 
2. Approximately 15 more authorizers could be considered close to meeting targets. Most of 
those authorizers had an equal or lower proportion of schools in the high/very high growth 
area compared to their proportion of schools/students in the typical/average growth category. 
Only the 22 authorizers meeting targets for Indicator 1 and Indicator 2 were included in 
subsequent analyses. 


Of the 22 authorizers meeting student academic targets, 12 failed to meet targets on the 
remaining nine indicators. In some cases, authorizers were removed from consideration for 
failing to meet one of the remaining indicators, but more often authorizers that failed to meet 
one indicator also failed to meet others. 
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Thus, 10 authorizers were deemed to have met targets on evaluated indicators and formed 
the group of authorizers in the strong performance group. Following the advice of the 
Advisory Panel, the selection of the initial group of five authorizers was based on several 
factors including generalizability by authorizer type, portfolio size, and area of the country. 
The pool of 10 potential authorizers contained a disproportionate number of authorizers from 
the east. Thus, while the project was able to achieve significant diversity in type and portfolio 
size, the first five research group authorizers are not widely spread across the country: three 
are from the East, one from the Midwest, and one from the South. 


Portfolio size : ; 
Type Geographic Region 
C# of Schools) 


Authorizer 1 111 ICB East 
Authorizer 2 11 NFP Midwest 
Authorizer 3 81 SEA East 
Authorizer 4 25 LEA South 
Authorizer 5 125 HEI East 


Note: ICB: Independent Chartering Board; NFP: Not-For-Profit; SEA: State Education Agency; LEA: Local 
Education Agency (school district); HEI: Higher Education Institution. The number of schools in each authorizer’s 
portfolio is from the 2015-16 academic year. 


AUTHORIZER SELECTION METHOD: MODERATE PERFORMANCE GROUP 


The group of authorizers with moderate performance was selected based on two criteria: 


e Similarity to strong performance group authorizers. The set of moderate performance 
group authorizers were selected to be as similar as possible as the strong 
performance group in terms of size, type, and region of the country. 


e Moderate, but not weak, academic outcomes. We intentionally looked for and selected 
authorizers whose academic Indicators 1 and 2 can be characterized as less strong 
relative to the strong performance group, but also not weak. Generally, that included 
authorizers with a relatively high percentage of schools making “typical” academic 
growth/proficiency, and that had between 30-40% of their portfolio with low growth/ 
proficiency. The purpose of selecting authorizes with moderate academic performance 
was to increase the precision and applicability of recommendations made to the field. 
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Four moderate performance group authorizers met these expectations, were willing to 
participate, and were thus included in the study. 


Portfolio size . F 
Type Geographic Region 
C# of Schools) 


Authorizer 1 32 ICB South 
Authorizer 2 24 SEA East 
Authorizer 3 49 NFP Midwest 
Authorizer 4 13 HEl Midwest 


CONTENT & CASE STUDY PROCESS SUMMARY 


After selecting authorizers for inclusion in the study, a deep investigation of perspectives and 
practices ensued. Following the case study process as outlined by Yin (2015), researchers 
from NACSA and Public Impact engaged in a range of activities designed to provide a 
comprehensive description of the approach to authorizing, including: 


e Case Study Protocol: Building from the domains used by NACSA to evaluate the 
practices of authorizers as well as the advice of Advisory Panel members, researchers 
created a case study protocol and specific domains of inquiry. Key questions and 
domains of inquiry can be found in Appendix D. Each research team dyad was 
responsible for one of the four key domains of inquiry (e.g., Application Systems, 
Organizational Structure, Organizational Culture, and Performance Management), and 
explored that domain for both groups of authorizers. 


e¢ Document and Artifact Review: Researchers reviewed a range of documents and 
artifacts (available in Appendix D). These data were used both to describe authorizing 
practices and to more clearly focus individual interviews. 


e Interviews and Site Visits: Researchers spent two days at each QPP strong 
performance group site interviewing authorizers and other key stakeholders. The 
purpose of the interviews and site visits was to (a) get clarification on authorizing 
practices after examining documents and artifacts, and (b) more clearly understand 
how and why authorizers engage in specific practices. Individual and small group 
interviews were conducted at each site. Most interviews were with authorizers (e.g., 
day-to-day decision makers, board members), but researchers also interviewed other 
key stakeholders (e.g., school operators, charter support organizations) to deepen 
and triangulate data analysis. The same process was used for moderate performance 
group authorizers, although telephone interviews were used in lieu of on-site visits. 


KEY FINDING GENERATION AND VALIDATION PROCESS 


Case Study Reports for Strong Performance Group Authorizers 
Research team dyads wrote their respective section for the case study reports (e.g., dyad 
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responsible for Application Systems & Process wrote that section for each of the five case 
study reports of authorizers with strong outcomes). Those reports were shared and critiqued 
by all members of the research team. The project’s senior researcher lightly edited and 
modified each section for consistency in tone and approach, but initial research team dyads 
signed off that no substantive content was changed. Case study reports were also shared 
with key stakeholders at each of the five research group sites, asking for feedback (e.g., 
member check’). Edits were made from feedback received, and the reports were sent to 
those stakeholders a second time requesting any additional feedback before they were 
finalized. 


Findings Between and Across Groups 

Research team dyads drafted a written description of practices for strong performance group 
authorizers, relying on case study reports. Those teams then compared findings across the 
five strong performance group authorizers to identify similarities across that group. Those 
drafts were circulated and critiqued by all members of the research team. That process 
yielded a list of authorizer practices and contexts that appear to be similar across strong 
performance group authorizers. 


Research team dyads used those findings to create a set of hypotheses specific to practices 
that are similar across strong performance group authorizes. They then tested those 
hypotheses against findings from the moderate performance group of authorizers. That 
process resulted in a list of practices that appear to be (a) common across both groups 

of authorizers (hypothesis not supported because practices were also common across 
moderate performance group authorizers), and (b) unique to strong performance group 
authorizers (hypothesis supported because practices not found to be common across 
moderate performance group authorizers). 


Validation and Vetting 
Findings comparing the strong and moderate performance groups were shared with many 
key stakeholders for further vetting and validation. 


QPP Advisory Panel. The QPP Advisory Panel reviewed all findings during an in-person 
meeting. That meeting was used to engage the expert judgement of key authorizers and 
others in the field. In small and large groups, Advisory Panel members discussed findings 
they agreed with, findings they had questions about, findings they disagreed with, and 
findings they wanted to know more about. Four of the five strong performance group sites 
participated in this advisory panel meeting. 


NACSA Leaders Program. Authorizers comprising NACSA’s Leaders Program similarly 
engaged the key findings. Members of the Leaders Program reviewed key findings in 
advance, and in small and large groups provided feedback to the research team. 


Internal NACSA Staff. Content experts—primarily from NACSA’s Talent & Engagement and 
Authorizer Development Divisions—also provided feedback on the findings. During three 1.5 
hour meetings, internal staff provided specific and detailed feedback on key findings. 


Data from this internal vetting resulted in a slightly modified list of key findings, with enhanced 
external validity. In no instance were any findings changed without strong support of the data. 


‘A research technique also knows as informant feedback or respondent validation. 
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In most instances of question or disagreement—of which there were not many—participants 
sought additional clarity on what the practice looked like. Subsequently, the research team 
made changes to provide additional descriptions of that practice. In other cases, participants 
asked questions beyond the scope of the original QPP. Those questions will inform future 
authorizer quality research work NACSA will pursue. 


A Note About Causality 

The method and process described here has been used successfully in other educational 
and business research in identifying correlates of success. It does not, however, yield causal 
relationships. Thus, key findings from the QPP should be positioned as correlates of strong 
outcomes that likely are worthy of adoption, but additional work will be needed to understand 
any causal relationships. 
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C. PORTFOLIO AND PERFORMANCE OUTCOMES 


PORTFOLIO OUTCOMES 


1. Portfolio contains few academically poor-performing schools 


e Data Used: Academic growth data over the last three available testing years. 
Graduation rate? and state test score proficiency data was also used. 


e Method/Targets: No more than 30% of schools or students in the authorizer’s 
portfolio were in low or very low growth’ categories in more than one academic year 
across English/Language Arts and Mathematics. In addition, no more than 10% of 
schools in the authorizer’s portfolio were in the bottom 5% of the state’s proficiency 
distribution across English/Language Arts and Mathematics. 


e Additional Information: As growth and proficiency information and distributions 
within each state were used, authorizer portfolio analysis on this indicator is relative 
to the performance of other schools in that state. Thus, “poor-performing” schools are 
relative to the growth and/or proficiency distribution within that state, not across states. 
In addition, while most states used student growth percentiles, states employed varied 
methods to assess academic growth (e.g., value-added gain index). In each case, 
the analysis uses the state’s method of assessing growth and evaluates authorizer 
portfolios relative to the state’s definitions. Descriptions of how each state assesses 
growth and state targets for very high, high, typical, low and very low growth/ 
proficiency categories are available upon request. 


2. Portfolio contains many academically high-performing schools 


e Data Used: Academic growth data over last three available testing years. Graduation 
rate and state test score proficiency data also used. 


e Method/Targets: More schools and students in “high” or “very high” growth 
categories compared to schools and students in “typical” growth categories in more 
than two academic year across English/Language Arts and Mathematics. 


e Additional Information: Like Indicator 1, growth and proficiency information and 
distributions within each state were used. Consequently, authorizer portfolio analysis 
on this indicator is relative to the performance of other schools in that state, not across 
states. 


3. Portfolio contains schools that are financially viable 
e Data Used: Authorizer reported financial data for each charter school in their portfolio. 


e¢ Method/Targets: Examination was customized to financial data collected and 
reported by each authorizer. Data were examined across three years. Both short- 
and long-term financial data was analyzed, typically including school financial audits 
(especially uncorrected/continuing audit findings), debt/income ratio, debt/asset ratio, 


?Four authorizers had a significant proportion of students and schools in middle/high school grades (60% or more students/schools in middle/high 
schools), making examination of their graduation rates critical in describing the overall academic performance of the authorizer (i.e., enough students/ 
schools that would far outweigh test score data typically available for grades 3-10). In each case, the graduation rate data did not demonstrably 

differ from conclusions drawn from state test score growth and/or proficiency information. None of those four authorizers were included in the strong 
performance group. 


3 Some authorizers were evaluated on a proficiency-only basis due to lack of growth data availability, as described earlier in this document. The same 
method and targets were used for authorizers evaluated via growth or proficiency. 
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and cash flow. On this indicator, 90% or more of schools in the authorizer’s portfolio 
were deemed to be financially viable to meet inclusion targets. Targets for each 
indicator are available upon request. 


¢ Additional Information: Some slight modifications were made for new charter 
schools to be deemed financially viable. New charter schools that had not yet been in 
operation for a full academic year did not have an audit, so other measures were more 
heavily scrutinized (e.g., cash flow). 


4. Portfolio contains schools that are fully accessible to all students 


e Data Used: Publicly available data on student enrollment for the most recently 
available academic year. Data examined included the proportion of students of color, 
students living in low-income households (proportion of students receiving free or 
reduced lunch), students with disabilities, and English Learners. Only data at the 
authorizer portfolio level (aggregated information) was used in the comparisons. 


e Method: A comparison group of non-charter schools was selected for each evaluated 
authorizer. Generally, for authorizers with statewide chartering authority, all non- 
charter schools in the state were selected as the comparison group. For authorizers 
with geographically limited authorizing authority, non-charter schools in that same 
geography formed the comparison group (i.e., a district authorizer data was compared 
to non-charter data in that same district). In the few cases, an authorizer had a more 
expansive chartering authority but had the vast majority of schools in a particular 
geography (e.g. statewide authorizer but with 90% of schools and students in one 
district), that particular geography was used as the comparison group. 


e¢ Targets: No more than 10 percentage points below the comparison group in two 
evaluated areas (students of color and low income), and no more than five percentage 
points below the comparison group for English Learners. 


e¢ Additional Information: Consistent with guidance given by the Advisory Panel, no 
upper limit (i.e., aggregated data on any indicator above the comparison group) was 
set. Eight of the 22 authorizers meeting targets for Indicators 1 and 2 did not meet 
targets for this indicator, with discrepancies ranging from mild (11 percentage points 
below comparison for free/reduced lunch proportions) to moderate (18 percentage 
points below comparison for free/reduced lunch). Of note, the majority of authorizer 
portfolios had higher proportions of students of color and lower income students than 
their comparison group, and in many cases much higher (20 to 30 percentage points 
higher in a number of cases). Discrepant rates for English Learners was rare, with 
virtually all evaluated authorizers having rates within 3 or 4 percentage points of their 
comparison group. No targets were set for the proportion of students with disabilities, 
given that the Advisory Panel deemed proportions of students with disabilities to 
be heavily influenced by school decisions. Data on students with disabilities was 
collected and analyzed for egregiously low rates that could have informed final 
selection of authorizers as a “tie-breaker.” In practice, however, no authorizer meeting 
performance indicators 1 and 2 had more than a six-percentage point difference 
than their comparison group, and among the five authorizers selected for the strong- 
performance group, none had more than a four-percentage point variance from their 
respective comparison group. 
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5. Portfolio contains schools that manage resources ethically 


e Data Used: Publicly available data (via internet searches) on instances of fraud, 
conflict of interest, cheating, or other unethical use of financial or human capital 
resources. 


¢ Method/Targets: Internet searches were conducted for any/all instances of unethical 
behavior among schools in an authorizer’s portfolio. Searches were conducted using 
the following descriptors: “fraud”, “corruption”, “financial mismanagement”, “conflict 
of interest”, and “cheating”. Searches were conducted with each of these terms and 
with the names of each school in the authorizer’s portfolio. Data was collected and 
analyzed going back three years. A qualitative analysis of the collection of unethical 
behavior in the portfolio was conducted that included the aggregate number of unique 
instances of unethical behavior to determine the degree to which there was a pattern 
of widespread unethical behavior in the portfolio (rubric is available upon request). 
Generally, authorizers with multiple instances of the same kind of unethical behavior 
by different schools, or more than three instances of unethical behavior were deemed 
to not meet the target for this indicator. 


e¢ Additional Information: Very few instances of unethical behavior were uncovered 
among authorizers meeting performance indicators 1 and 2. Two of those authorizers 
were deemed to have widespread unethical behavior in their portfolio (one had 
multiple instances of financial fraud across different schools, and another had multiple 
instances of a host of unethical behavior issues across a number of schools operated 
by a single management organization) and were excluded from consideration. 


6. Performance of individual schools is accurate, transparent, and 


widely accessible to interested stakeholders 


e Data Used: Publicly available authorizer- or state-produced school performance 
reports. 


e Method/Targets: Internet searches for school performance reports were conducted. 
Most were found on the websites of state departments of education and the 
authorizers themselves. Examination of the content of those reports was conducted. 
Reports had to include data for every school in the authorizer’s portfolio in the 
following areas (a) academic/student outcome data, (b) financial performance data, 
and (c) organizational data (e.g., enrollment, socio-demographics). The availability 
of individual school data in those three categories at least twice over the last three 
school years was deemed sufficient to meet the inclusion target. 


e Additional Information: Eight authorizers meeting indicator criteria 1 and 2 did not 
have school-level financial information publicly available (either authorizer produced or 
state produced). 


7. Portfolio contains schools that have full autonomy within the 
bounds of federal/state law 


e Data Used: State policy (charter school governance provisions) and authorizer policy 
on school governance. 


e Method/Targets: A comprehensive understanding of how authorizers extend 
autonomy to schools was examined as a part of the case study process and is 
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thus considered a practice rather than an outcome. However, minimum autonomy 
standards as articulated in state law and authorizer policy were examined to determine 
the degree to which authorizers had and exercised direct control over school 
governing boards. Authorizers were excluded if they appointed individual charter 
school governing boards or if the school district’s governing board also served as the 
governing board for individual charter schools. 


e¢ Additional information: Among authorizers meeting Indicators 1 and 2, two district 
authorizers were excluded because their elected governing board also served as the 
governing board for charter schools in their portfolio. 


PERFORMANCE OUTCOMES 


8. Open strong schools 
¢ Data Used: State reported data on school openings and closings. 


e Method/Targets: Examination of schools in each authorizer’s portfolio that opened 
and closed prior to that school’s 2"? year serving students (e.g., a school that opened 
and served students in the fall of 2013 but subsequently closed any time before the 
beginning of the 2014-15 school year is deemed a 1* year closure). Any instance of a 
1st year closure in the authorizer’s portfolio over the last two years excluded them from 
consideration. 


e¢ Additional Information: Four authorizers meeting Indicator 1 and 2 targets were 
excluded from consideration due to first year closures. 


9. Revoke/close charters for egregious operational, financial, or 


unlawful practices 


e Data Used: Publicly available data (via internet searches) on instances of fraud, 
conflict of interest, cheating, or other unethical use of financial or human capital 
resources (see Indicator 5 for complete description). In some instances, cross- 
references identified schools with state reported databases on charter school 
closures. 


e Methods/Targets: Examined degree to which schools identified with unethical and 
unlawful behavior were closed. Target is for all schools found to have egregious 
negative practices to have subsequently closed. 


e Additional Information: Of the two authorizers deemed to have egregious unethical/ 
unlawful behavior among schools in their portfolio over the last three years, one was 
excluded because those two schools had not closed two years post-infraction. In those 
instances, the school was restructured with some new board members and leadership, 
but not closed. 


10. Close schools that fail to meet rigorous academic 
performance standards 
e Data Used: Academic proficiency data and state data on closures. 


e Method/Targets: Examination of schools in the bottom 5% of the states’ proficiency 
distribution of English/Language Arts or Mathematics. Authorizers were deemed to 
meet these criteria if (a) they have no schools in the bottom 5% of proficiency over the 
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last three years or (b) they have closed at least one school in the bottom 5% over the 
last three years. 


e Additional Information: Only schools meeting targets for Indicators 1 and 2 were 
evaluated on this Indicator; thus, there were very few authorizers with schools in the 
bottom 5% of their state’s proficiency distribution. There were three authorizers that 
appeared to have a relatively large number of schools (3-5) in the bottom 5% that 
(a) had been open for more than five years, (b) were not classified as an alternative 
education campus, and (c) had not yet closed. Those three authorizers failed to 
meet targets set in other indicators so no follow-up was conducted to ascertain any 
legitimate reason those schools remained open. 


11. High-performing schools/operators expand to serve more students 


e Data Used: Student performance growth/proficiency data and school-level enrollment 
over the last three available testing years. 


e Method/Targets: Inspection of enrollment changes (over a three-year period) ina 
random sample of schools in each authorizer’s portfolio that met criteria for “high” 
or “very high” growth (or proficiency for authorizers in states with no-growth data) 
in two of three years in either English/Language Arts or Mathematics. Any increase 
in enrollment among 75% of sampled schools was deemed sufficient to meet this 
indicator. 


¢ Additional Information: Virtually all (90% or more) of sampled schools showed 
some growth in enrollment. Visual inspection of schools with average growth showed 
similar growth patterns as high-growth schools. Thus, while high-growth schools were 
expanding to serve more students, so were average-growth schools. Unfortunately, 
it was very difficult to ascertain schools that were replications of existing high- 
performing schools and thus only enrollment growth among high performers was used 
for this indicator. 
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D. Interview & Document Review Protocol 


(Distributed to participating authorizers)* 


DOMAIN 1: ORGANIZATIONAL PURPOSE & CULTURE 


Area Description 
This domain will describe your unique approach to authorizing and your organizational 
culture. We will want to explore four primary areas with you and staff including how you: 


e Approach authorizing generally; 

e Execute authorizing consistent with your mission; 

e Think about growth and type of schools in your portfolio; 
e Extend autonomy to schools; and 


e Manage systems related to equity issues. 


Suggested Interviewees 


e Authorizer/School Board Member (Note: we will also be asking him/her questions in 
other domains) 


¢ Superintendent or Equivalent — if applicable (Note: we will also be asking him/her 
questions in other domains) 


e Executive Director/Authorizing Staff Leader 


e Lead staff member that works on compliance and accountability functions with schools 


Documents Reviewed 

e¢ Most recent strategic plan 

e Mission statement 

e Any descriptions of special/unique practices for the following areas: 
» Special Education 
» English Language Learners 
» School Discipline 
» Backfill 


»  School-level racial/ethnic and/or socioeconomic diversity 
Interview Questions 


Purpose/Approach 
e What is your organization’s purpose for authorizing? 


e Would you describe your approach to authorizing decision making to be more focused 
on creating strong processes for making decisions, or more focused on utilizing the 
professional judgment of staff and other experts? 


4At each site, we also interviewed other relevant stakeholders. These stakeholders typically included school/network leaders, charter association or 
other education reform community leaders, and others noted as important to the charter sector in that locale. Interview questions were customized for 
those audiences at each site from document reviews, information acquired from authorizers, and knowledge of that organization. 
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e How would you describe your district’s views on authorizing? 


¢ How much attention do you pay to the overall success of charter schools across the 
country? 


¢ To what degree do you think that charter schools are/should be: 
» A replacement to the district-run sector 
» A hub of innovation for the district-run sector and public education writ large 
» Acompetitive force for change and improvement in both sectors 
» A set of schools that fill in gaps in local district programs/schools 


» Akin to local district programs/schools (i.e., like magnet schools) 


Portfolio Growth 
¢ How important is growing the number of schools in your portfolio? 


¢ How much do waiting list numbers from your existing schools factor into decisions to 
charter new schools? 


e Would you describe your office’s approach to new schools to be more (a) proactively 
seeking out and recruiting new schools, or (b) open to new schools that apply to your 
office? 

Management Organizations 


e Does your organization have a preference for “independent” schools compared to 
“managed” (CMO- or EMO-run) schools? If so, what and why? 


e In recent years, what types of applications have you received more of and why do you 
think that is? 


e How do you incorporate (or perhaps define) the needs of the local community in your 
decision-making processes? 
Autonomy 


e¢ To what degree do you think charter schools should have complete autonomy in how 
they spend their money, decide on staff, and operate their educational program? 


e What areas do you think it’s OK/necessary to limit charter school autonomy and why? 


Equity 
e How do you think about—and what, if anything, do you do relative to---the following 
areas of student/school equity? 


» Special Education 

» English Language Learners 
» Discipline 

» Backfill 


» School diversity 
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DOMAIN 2: ORGANIZATIONAL CAPACITY 


Area Description 
This domain will describe your staffing, planning, and development processes. We will want 
to explore four primary areas with you and staff including how you: 


¢ Set goals and establish strategic priorities; 
e Create and execute strategic planning; 
¢ Conduct human capital identification and development; and 


e Establish relationships with organizations/entities outside of your authorizing office. 


Suggested Interviewees 
e Executive Director/Authorizing Staff Leader 
e Staff that lead any part of your human capital system 
e Staff that lead relationship development/management with non-charter school external 
entities (e.g., other education reform organizations, community organizations, etc.) 
Documents Reviewed 
¢ Organizational chart (showing titles and lines of reporting, if possible) 


¢ Organizational budget 
Interview Questions 


Goal Setting & Strategic Priorities 


e What do you think is exceptional about how your organization is structured that 
contributes to strong performance? 


e¢ How do you determine success in authorizing? 


e¢ How do you share (internally and externally) success and challenges? 


Strategic Planning 
¢ How does your office do strategic planning (if at all)? 


e¢ How do you align resources to achieving strategic ends (including use of external 
consultants, if applicable)? 


e What are your sources of support (human capital, financial, etc.) to achieve 
organizational goals? 
Human Capital Identification & Development 


e How has your office staffing structure changed, and how do you see it changing over 
the next five years? 


e How do you hire new staff (i.e., talent identification and selection)? 
¢ How do you do staff professional development, retention, promotion, dismissal, and 
performance feedback? 
Relationships with Entities Outside of Authorizing Office 
¢ How involved are you with (and why/why not): 


» Other Internal Divisions 
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» State policymaking/makers 
» Education reform organizations 
» Community organizations 


» Other authorizers with schools in your market (if applicable) 


DOMAIN 3: APPLICATION SYSTEM AND PROCESS 


Area Description 
This domain will describe your application system and process. We will want to explore four 
primary areas with you and staff including how you: 


e Develop your application materials and other application-related documents; 
e Develop and execute criteria for new school applications; 
e Evaluate new charter school applications; and 


e Interact with your Board on the application process. 


Suggested Interviewees 
e Staff responsible for leading any/all part(s) of the application process 


e Superintendent or equivalent, if applicable (can combine questions in this domain 
during a single interview time slot with the Superintendent) 


¢ School/Authorizing Board Member (can combine questions in this domain during 
single interview time slot with school board member) 
Documents Reviewed 
e Example(s) of a Request For Proposals (if applicable) 
e Application Packet 
e Evaluation criteria and rubric 


e Sample applications received and decisions made (this can be a simple list of the 
applicant name, year applied, and if they were approved, declined, or pending) 


e Applicant Interview process and rubric 


e One example of a recent completed application and authorizer evaluation products 
(e.g., completed application, scoring rubrics, and/or reviewer analysis) 


e¢ Example of recommendations and any supporting materials presented by staff to the 
board in preparation for board decisions 


e¢ Examples of any notice provided to applicants regarding approval or denial decisions 
Interview Questions 
Application Materials and Process 

e How do you identify demand/need for new charter schools? 


e What guidance and materials does your office provide charter school applicants? 


e What parts of your application process and systems are public? 
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Application Criteria 
e What are your requirements and criteria for: 
» Educational program 
» Business/Financial plan 
» Operational plan 
» School leadership 


e Do you have specialized criteria for specific types of applicants beyond traditional 
start-ups (i.e., existing operators, alternative education schools, virtual school 
applicants, applicants planning to contract with a management organization)? 


e If your application process has several stages (e.g., LOI, prospectus, final application), 
how do you manage that process and how are decisions made? 
Application Evaluation 
e What, in detail, is the process used from application receipt to final decision? 
e How do does your office run its charter application interview process? 


e¢ Does your office use external reviewers to evaluate charter applications? If so, how did 
you find them and how do you use them? 


e How are recommendations for approval or denial reached? 


Authorizing Board Responsibilities 


e How does the Superintendent (or equivalent) and Authorizer/School Board participate 
in the application process? 


e What products are provided to the Superintendent (or equivalent) and/or Authorizer/ 
School Board (written report, verbal summary, etc.)? 


e How are approval/denial decisions reached at the board level? 


e What is the form and extent of your final evaluation decision? (E.g., Are rubrics/notes 
provided to the applicant? Is a summary provided? How are decisions reported to the 
public?) 


DOMAIN 4: PERFORMANCE MANAGEMENT 


Area Description 
This domain will describe your approach to oversight and accountability. We will want to 
explore the following areas with you including how you: 


e Execute your pre-opening systems and processes; 

e Use your performance framework; 

e Conduct general oversight of schools in your portfolio; 
e Execute your renewal and revocation systems; and 


e Think about and execute systems related to replication and growth. 


Suggested Interviewees 


e Executive Director/Authorizing Staff Leader 
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e Staff responsible for any/all parts of performance management, including pre-opening, 
compliance, accountability, renewal, contracting, and revocation systems 


e¢ Superintendent or equivalent, if applicable (will combine questions in this domain 
during a single interview time slot with the Superintendent) 


¢ School/Authorizing Board Member (will combine questions in this domain during a 
single interview time slot with a school board member) 
Documents Reviewed 
e Example of a fully executed charter contract 
e Pre-opening checklist 
e¢ School Performance Framework 
e Accountability Compact 


e Master calendar of reporting requirements or similar document (e.g., description of 
required academic, financial, and legal/organizational reporting by schools) 


e Intervention system description, particularly under what conditions you decide to 
intervene and the nature of that intervention 


e Site visit process (if you conduct them), including protocol, rubric, and other supporting 
information 


e Most recent accountability report (or ways you make school data public) 

¢ Sample of a charter school annual report required by the authorizer (if applicable) 
e Renewal process 

e Revocation policy and process 

e School Closure and “wind down” policy 


e Replication and/or extension policies (if applicable) 
Interview Questions 


Pre-Opening Systems and Practices 
e Please describe your system from application approval to first day of school opening 


e¢ What autonomy rights/responsibilities are clearly laid out in the charter contract? 


Performance Framework 
e What are your performance expectations for schools in: 
» Academics 
» Finances 
» Operations 


e¢ How do you make clear what those expectations are? 


Performance Accountability 
e What systems do you use for monitoring school performance expectations? 


e Describe your intervention system including how you determine the need to 
intervene—and what are your intervention practices? 
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e What kind of technical assistance does your office provide in executing your 
intervention system? 


e What is your process for utilizing site visits to monitor and evaluate school 
performance and compliance? How valuable do you find site visits (if conducted) and 
why? 


¢ How do you communicate to schools and the public regarding how schools are 
performing? 


e What and how do you monitor compliance with charter law and policies that fall 
outside of your performance framework? 
Extension, Renewal, and Revocation 
e What are your standards for extensions and renewals? 
e Please describe what you do when a decision for non-renewal is made. 


e What is your process for revocation? 


Replication and Growth 


e¢ What (if any) policies and practices do you have to guide decisions for replicating and 
expanding charter schools in your portfolio? 


e How actively do you encourage the replication and expansion of quality schools? 
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